A Fixed Size Storage O(n3) Time Complexity Learning Algorithm for Fully Recurrent Continually Running Networks

There are two basic methods for performing steepest descent in fully recurrent networks with n noninput units and m = O(n) input units. Backpropagation through time (BPTT) [e.g., Williams and Peng (1990)l requires potentially unlimited storage in proportion to the length of the longest training sequence but needs only O(n’) computations per time step. BPTT is the method of choice if training sequences are known to have less than O(n) time steps. For training sequences involving many more time steps than n, for training sequences of unknown length, and for on-line learning in general one would like to have an algorithm with upper bounds for storage and for computations required per time step. Such an algorithm is the RTRL algorithm (Robinson and .Fallside 1987; Williams and Zipser 1989). It requires only fixed-size storage of the order O(n3) but is computationally expensive: It requires O(n4) operations per time step.’ The algorithm described herein2 requires O(n3) storage, too. Every O(n) time steps it requires O(n4) operations, but on all other time steps it requires only O(n2) operations. This cuts the average time complexity per time step to o(n3).